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ABSTRACT 

Instruction of large groups of students in the use of 
audiovisual (AV) equipment presents a problem when the students* 
performance can only be evaluated by individual examinations on 
actual equipment. An attempt was made, therefore, to develop an 
objective, paper-and- pencil test which would provide the same 
evaluation. Although students in a basic instructional materials 
course improved their scores on the objective test between pretest 
and posttest, the correlations between scores on the objective test 
and a performance test were very low, even though the correlation was 
significant for female subjects considered alone. While this attempt 
to develop an objective test of competence with audiovisual equipment 
failed, future research Knight be more successful if care is taken to 
preserve the normal motivation of the student subjects. (RH) 
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EXPERIMENTAL DESIGN OF AN OBJECTIVE VERBAL-PICTORIAL TEST 
FOR AUDIOVISUAL EQUIPIiENT OPERATIONS SKILLS 



PROBLEM 

The object of all education is to affect some change in students. To 
justify education it is necessary to evaluate students in relation to 
stated objectives. Paper and pencil tests are assumed to be adequate 
in most academic situations for testing conceptual ability. However, 
to evaluate a psychomotor ski l l it has traditionally been considered 
necessary to test by observing the performance of the skill. The 
project rep or tec* here was an attempt to test the validity of this 
assumption. 

A basic Instructional Materials course was required for Illinois teacher 
certification. Consequently all education majors enrolled in the basic 
integrated IM-AV course. One of the objectives of this course was to 
have the students become reasonably proficient in the operation of the 
traditio na l audiovisual equipment. Between 250 and 325 students per 
quarter registered for this course which made the traditional method of 
machine operation instruction entirely inadequate. A special laboratory 
was equipped and staffed so that students could learn this skill as an 
outside assignment. Time was provided for additional practice when 
necessary with a trained machine instructor available to give individual 
help. This arrangement proved quite satisfactory for the students because 
it gave them flexibility and adequate time to learn the task to an estab- 
lished criterion. From the instructor’s point of view however, it was 
found that a great deal of time was devoted to the administration of the 
criterion test. A need existed therefore, for an adequate, but less time 
consuming, test to substitute for the observer administered test. 

It was assumed that an adequate substitute te3t would have the following 
attributes and advantages: 

(1 ) Test administration would be standardized and not subject to 
the vagaries and personal feelings of an observer. 

( 2 ) Grading would be standardized so personality factors would 
be eliminated. 



Data for this study was collected while the author was on. the faculty of 
the Instructional Materials Department, College of Education, Southern 
Illinois University, Carbondale, Illinois, and was supported by a grant 
from the OFFICE OF RESEARCH AND PROJECTS of that institution. 

Data reported in this paper were processed at the University of Connecticut 
Computer Center and supported by NSF Grant GP-1 81 9« 
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(3) Large groups of students could ')& given the test simultaneously, 
thus releasing the instructor for other tasks, 

(4) The test could be used to identify individuals who already 
possess adequate machine operation skill and therefore do 
not need training* 

It was hypothesized that the operation of a motion picture projector, or 
any machine, depended in paro on perception* It was further hypothesized 
that if the student was able to operate the equipment he would be able to 
recognize or perceive from a photograph the critical points at which the 
film and the machine were inconsistent. The purpose of this experimental 
design was to test the hypothesis that there would be a correlation between 
a student * s ability to perceive a correctly functioning machine from an 
iconic representation and his score on a performance test* A 16mm motion 
picture projector was chosen for use because it is perceptually the most 
complex piece of audiovisual equipment* In addition this was a pilot study 
to explore the problems involved in producing such a test. 



RELATED RESEARCH 



Cronbach (2) cites a study by the Navy in which one of two forms of a 
test (words alone and pictures supplemented by words) was given to gun 
crews to evaluate the training of gunners* The questions were related 
to parts of the guns, duties of the crew, appearance of tracer paths and 
other items which could be visualized* Pictorial test results were found 
to have a correlation of .90 with the instructors* marks bared on gun 
operations while the correlation of the verbal test with instructors* 
marks was only *62. It would seem probable from this study that a 
pictorial test could be constructed which would adequately correlate 
with an outside criterion* Questions here dealt witu items which could 
be visualized and the results indicated that words alone were inadequate 
to elicit the desired responses. 

W illiams (5) on the other hand found that neither the verbal nor the 
verbal-pictorial test used in his study correlated better with the out— 
side criterion the instructor assigned rank* He did report however that 
the verbal-pictorial test gave significantly higher scores* The outside 
criterion in this case was the quality of products produced by the 
student rather than ability to use the equipment properly. It is 
possible to speculate that the test and the criterion required different 
abilities. 

Brown and DeKieffer (1 ) report in a more general article that their 
technique for presenting a biology examination by photographic slides 
had seven different advantages. Their prime interest was that the slide 
test overcame unsatisfactory features of usual biology examinations 
where students are required to move from station to station. Of most 
importance to this study is their report that the flexibility and ease 
of administering the test were much greater than the traditional test* 
They found the use of slides in biology examinations had advantages for 
both student and instructor. 
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Lefkowith (3 ) fourxi that as the ioonioity level of the pictorial stimuli 
used to test came closer to the iconicity level of the pictorial stimuli 
used to teach, scores on the tests increased# He also noted that as 
pictorial stimuli used to test became more iconic the correlation between 
the subjects* scores on that method and their scores on an equipment test 
became hitler. In addition, he stated it was demonstrated empirically 
that the pictorial tests were both valid and reliable enough to be used 
in actual testing programs. This supports the hypothesis that a properly 
constructed visual test is a practical possibility# 

Thelen (4) hypothesized that using motion pictures to test a time dimension 
and using stereoscopic and color pictures in some cases would increase the 
realism and therefore validity of a test. While this is an interesting 
supposition, for purposes of this study it was felt that it would be 
im practical to administer projected group tests using color slides because 
of crowded classrooms. Since this study was an attempt to demonstrate the 
practicability of a testing method it was decided that colored photographs 
would be too expensive to print in quantity. Therefore black and white 
printed photographs were used. 

Although research related to pictorial testing is meager and the re suit s_ 
are unclear, it appeared possible to construct a practical pictorial test 
which would eliminate the time consuming observer rating technique 
generally employed to test psychomotor ability. TVo hypotheses were 

selected for study. 

Hypothesis 1 ; The correlation between the verbal-pictorial paper 

and pencil test will correlate .70 or better with 
a check sheet criterion test. 

Hypothesis 2: There will be a significant difference between the 

means of the verbal-pictorial paper and pencil 
test scores when administered as a pre-test and 
as a post-test to the same individuals. 



METHOD 

An operations check sheet (Appendix A) was constructed to yield a score 
which would reflect the ability of student subjects to operate a Bell and 
Howell Model 385 motion picture projector. Each item in the process was 
weighted so the final check sheet was sensitive to major and minor errors 
in the operation process. 

A series of photographs were then taken of the Bell and Howell 385 and a 
pilot study was conducted to select the types of items to be used in the 
final pictorial test. Samples of the chosen items are shown in Appendix 
B. Sample item No. 1 required the subject to indicate if the picture 
illustrated a correctly functioning sound motion picture projector. Four 
answers were possible for this item. Three answers required the subject 
to indicate if the projector was improperly threaded, improperly adjusted, 
or both improperly threaded and adjusted. Sample item No. 2 required the 
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student to indicate if the three pictured operations were presented in the 
proper sequence. The answer was either yes or no* Sample item No. 3 
required the student to indicate if the projector was properly threaded 
and adjusted to rewind a film. This question was also answered yes or no. 

The answers were written on the answer sheet (Appendix C) which wpf 
arranged so that the answers and other information could be coded on 
punch cards for analysis. Part IV of the test required the subject to 
write the names of the parts indicated. This part of the test was not 
directly related to the main study. The data was collected however to 
be analyzed to find how much incidental learning of nomenclature occurred 
as a result of the training. The experiment was conducted over a number 
of school terms. Data for some 250 individuals was collected. The test 
was given as a pre-test and as a post-test to approximately 110 subjects 
of which 98 proved usable. 

The pictorial test was timed for Parts I, II and III on the assumption 
that the range of scores would indicate both the accuracy and speed of 
perception of the subjects. The time for Part I was six minutes* for 
Part II three minutes* and for Part III two minutes. Part IV was not 
timed. The subjects were given the option of writing a descriptive 
term if they did not know the technical name. 

After pre-testing, students were given the criterion check sheet and 
trained with the understanding that the check sheet would be used for 
their final performance test. The subjects therefore knew . their 
behavioral objectives. Subjects were scheduled for their individual 
performance tests within a period of one week. As close as practical 
to the middle of the week in which the criterion performance test was 
being administered* the pictorial test was given as a group test within 
the regular classroom. 



TABLE I 

CORRELATION BETWEEN WHITTEN VERBAL-PICTORIAL (VP) TEST SCORES 
AND PERFORMANCE (P) TEST SCORES 





N 


X (VP) 


•=\(W) 


X (P) 


<=Mp) 


r 


Males 


30 


42.50 


5.96 


77.73 


5.11 


-.02 


Females 


68 


40.54 


6.30 


76.41 


4.51 


.22 


Total 


98 


41.14 


6.23 


76.96 


4.70 


.16 
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The correlation between the written verbal -pictorial test scores and the 
performance test scores is *16 for 98 subjects. Hypothesis 1 must there 
fore be rejected, A r of ,16 is not significant at the ,1 level for 96 
(N-2) degrees of freedom. Considering females only, however, the r is 
significant beyond the .05 level. 

Hypothesis 2 was stated as follows : There will be a significant differ- 

ence between the means of the verbal-pictorial paper and pencil test 
scores when administered ac a pre-test and as a post-test to the same 
individuals. Table II gives the means, standard deviations and t for 
98 subjects pre-test and post-test scores. 



TABLE II 

CORRELATED t TEST BETWEEN THE MEANS OF THE PRE-TEST (PRT) 

AND POST-TEST (POT) SCORES 





N 


X (PRT) 


^ (PRT) 


X (POT) 


«*>(P0T) 


t 


Males 


30 


30.33 


6.73 


; 42.50 


5.96 


10.10 


Females 


68 


27.71 


6.47 


40.51 


6.30 


14.38 


Total 


98 


28.51 


6.63 


41.14 


6.23 


17.59 



The t test between the means of the pre-test and post-test scores is 17*59 
for 98 subjects. This is significant beyond the .001 level. Hypothesis 
2 is therefore retained. 



SUMMARY 



In some instances, large numbers of students are required to learn how 
to operate basic audiovisual equipment. Because of the great amount of 
time needed to evaluate each student individually the need for a group 
test is indicated. A verbal-pictorial test based on the Bell and Howell 
Model 385 motion picture projector was developed as a pilot project to 
determine the feasibility of such an undertaking. Black and white still 
photographs, printed in booklet form, were used for the test. Students 
vere verbally cued to a pictorial test which required them to perceive 
the projector as either threaded or adjusted correctly. Additionally 
they were required to visually identify the correct sequence for operating 
the projector. They were also required, if possible, to list the nomen- 
clature of the most important parts of the projector. The verbal-pictorial 
test scores were correlated with scores on a criterion performance check 
sheet. Because the test was given as a pre-test and post-test to some 
subjects, it was also possible to calculate a correlated t test of the 
difference between the means. 
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The correlation between the verbal-pictorial test and the criterion 
check sheet scores was positive but low (r *= *1o;* If males and 
females ere considered separately the correlation is hi^ier for the 
females than for the males* Ihe t test between the means of the pre- 
test and the post-test yield a t of 17*59 (p^ *001)* 



CONCLUSIONS AND SPECULATIONS 



The conclusion which seems warranted fvom this study is that verbal - 
pictorial tests are not, at this time, a practical substitute for 
individual observer scoring of motion picture projector operation 
performance* This finding tends to follow Williams implied conclusion 
that the results of his verbal-pictorial test were unstable. The 
restalts are in variance with results of the Navy study described by 
Cioribach and the conclusions of Lefkowith* 

This writer speculates that moi ivation, or its lack, played a great 
part in the negative correlations between verbal-pictorial test scores 
and criterion performance test scores in the case of the male students. 
Subjects of this study were told they were taking part in an experiment 
arri that the results of the pictorial test would in no way affect their 
course grade* This is, on the face of it, the worst possible situation 
for motivation with college students. Additionally, scores on the 
criterion performance check sheet were probably compressed because the 
st ud ents knew they were required to receive a minimum score of 59 of 
the possible 80 or retake the test* 

The following recommendations are offered to researchers for further 
studies of pictorial testing procedures. 

1 . Item formats should be studied in more detail. 

2. Attention should be given to subject motivation* 

3. An attempt should be made to develop very sensitive criterion 
tests which will yield a wide range of scores. 

4. Time should be taken into consideration when weiring test 
results as this is a factor in efficient operation of equipment* 

5. Verbal-pictorial training should be given to subjects to 
determine if they can make acceptable scores on the verbal- 
pictorial test without having actually operated the equipment. 

6. The ability to transfer knowledge is most important to the 
educated teacher* Therefore, the ability to transfer the 
learning from one machine to another should be investigated. 
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APPENDIX A 



Southern Illinois University 
111 417 Laboratory 



3. 

4. 



5. 



6 . 



MOTION PICTURE PROJECTOR PROFICIENCY EXAMINATION 
Steps Completed in Pro .lection of Film 



1 • Clean pro jector (use brush) 

2. (a) Check forward-reverse switch 

(b) Check sound-silent switch 



Prefocns 

Turn motor and lamp on 
(in that order) 

Center and focus image on screen 

Check sound apparatus 
before projection 
Turn volume down to M 0 ,f after 
checking sound apparatus 
( leaving amplifier on ) 



Threading 

7. Thread film properly 

8 , (a) With correct loop length 

(b) Have film tight around 
sound drum 

(c) Advance film to title frame 
and stop 

•MMMWb 

Projecting 

9, Turn on machine, then turn on 
lamp at beginning of film 

10, Fine Focus (if necessary) 

11, Set volume at reasonable level 

12, Frame picture (if necessary) 

13 * # 10 , 11 , & 12 in sequence 

14. Turn off lamp at end of film 
(before motor is stopped) 

1 5. Turn off amplifier 

16. Rewind properly 

17. Unplug 



(4) Correct 


(1 ) Incorrect 


(0) Forgot 


(4) OK 




(0) Forgot 


(4) OK 




(0) Forgot 


(5) In order 




(0) Not in 






order 


(2) OK focus 




(0) Did not 






prefocus 


(4) OK 




(0) Forgot 


(4) OK 


(2) Sound up 


(0) Sound off 



(5) OK 


(0) Wrong 


(4) Loop correct 


(0) Loop 




incorrect 


(4) Film tight 


(0) Film loose 


(2) Correct 


(0) Forgot 


(4) OK 


(2) Light on too 




soon or too late 


(4) OK 


(1 ) Late focus (0) Out of 




focus 


(4) Volume OK 


(2) Too much/ (0) No volume 




Too little 


(4) OK 


(0) Not OK 


( 6 ) Correct order 


(0) Incorrect 


( 6 ) Correct order 


(2) Incorrect 


(4) OK 


(0) Forgot 


( 4 ) OK (1) Ask for help (0) Wrong 


(2) OK 


(0) Forgot 



TOTAL POINTS 



TOTAL TIME MINUS FILM TIME 



1 


1 


1 Too much time 


1 — 1 





20 18 16 
Point 8 
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appendix b 

Sample Items 



No. 1 




No. 2 
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APPENDIX B (Continued) 



No. 3 




Reverse view 
of 

takeup arm 




No. 4 








APPENDIX C 



BELL & HOWELL PROFICIENCY TEST ANSWER SHEET 

# * i 

Name S e x Class Standin g 

General Information 

1 . Number of hours practice on Boll and Howell 385 projector 

(Circle one) 0 1 2 3 4 5 or more 

2. Number of hours formal training on a Bell and Howell 385 projector. 

(Circle one) 0 1 2 3 4 5 or more 

3. Number of hours practice on any other projector, (specify) 

(Circle one) 0 1 2 3 4 5 or more 



PART I 

Sample 



1 






2 




12 




-23 




2 




13 




24 




4 




14 




23 




5 




01 




26 




6 




16 




27 




7 




17 




28 




8 




18 




-29 




9 




-11 




30 




19 




20 




31 




11 




21 




32 




22 




33 





PART II 

Sample 



40 






^1 




49 




42 




50 




43 




51 




44 












5? 




46 




54 




47 




51_ 




48 
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61 




6? 




63 
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65 
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76 82. 

77 83. 
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